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PREFACE 


This  report  was  created  for  the  P-16  Aircrew  Training 
Development  Project  contract  number  F02604-79-C8875  for  Tactical 
Air  Command  to  support  CDRL  no.  B069.  The  project  entailed  the 
design  and  development  of  an  instructional  system  for  the  F-16 
RTU  and  instructor  pilots.  During  the  course  of  the  project  a 
series  of  development  reports  was  issued  describing  processes  and 
products.  A  list  of  those  reports  is  contained  on  the  next  page. 
The  user  is  referred  to  Report  No.  34,  A  Users  Guide  to  the  F-16 
Training  Development  Reports,  for  an  overview  and  explanation  of 
the  series,  and  Report  No.  35,  F-16  Final  Report,  for  an  overview 
of  the  Instructional  System  Development  Project. 


F-16  AIRCREW  TRAINING 
DEVELOPMENT  PROJECT  REPORTS 


Copies  of  these  reports  may  be  obtained  by  Meriting  the  Defense 
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EXECUTIVE  SUMMARY 


Ihe  F-16  Aircrew  Training  Developnent  Project  is  thoroughly 
documented  by  a  series  o£  35  individual  development  reports. 

This  report  describes  the  organization  of  the  report  series  and 
provides  a  one  page  executive ■ summer y  of  each  report.  While  the 
Mnal  report.  No.  35,  is  designed  to  summarize  the  four  year 
development  effort  in  a  single  document,  report  No.  34  is 
designed  to  index  the  report  series  for  those  readers  who  desire 
additional  information  concerning  a  particular  event  or  phase 
within  the  program.. 
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A  USER'S  GUIDE  TO  THE  F-1 6 
TRAINING  DEVELOPMENT  REPORTS 


1.0  ORGANIZATION  OP  THE  REPORT  SERIES 

The  35  Development  Reports  in  the  F-16  Report  series  fall 
into  the  following  five  general  categories:  (1)  problem 
analysis,  review,  and  feasibility  studies,  (2)  methodology 
reports,  (3)  data  base  docunents,  (4)  system  operating  manuals, 
and  (5)  summary  reports.  The  reports  in  each  category  are 
listed  below: 


Problem  analysis,  review,  and  feasilbility  studies. 

Program  Work  Plan  (#1) 

Previous  ISD  Program  Review  (#2) 

Data  Collection  and  Management  Forms  (#30) 

Review  of  Existing  F-16  Task  Anaylsis  (#4) 

Data  Automation  of  Task  and  Goal  Analysis 
Existing  System  Review  and  Recommendations  (#11) 

Management  System  Needs  and  Design  Concept  Analysis  (#12) 

Program/System  Constraints  Analysis  Report  (#15) 

A  Study  of  Media  Production  and  Reproduction  Options 
For  the  F-16  Project  (#16) 

Computer  Managed  Instruction  for  the  F-16 
Training  Program  (#17) 

Recommendations  for  F-16  Operational  Flight 
Trainer  (OFT)  Design  Improvements  (#22) 

F-16  Instructional  System  Design  Alternatives  (#27) 

F-16  Training  Media  Constraints  and  Limitations  (#33) 


Methodology  Reports 

Derivation,  Formatting,  and  Use  of  Criterion-Referenced 
Objectives  (CROs)  and  Criterion-Referenced  Tests  (CRTs) 
(#5) 

Task  Analysis  Methodology  Report  (#7) 
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Objectives  Hierarchy  Analysis  I'ie t-nodology  Report  (#8) 
F-16  Media  Selection  and  Utilization  Plan  Report  (#20) 
F-15  Instructiona  Sequencing  Plan  Report  (#23) 

F-16  Training  System  Media  Report  (#30) 

F-16  Training  Media  Mix  (#31) 

F-16  Training  Media  Support  Requirements  (#32) 
Data-3ased  Documents 

F-16  TasK  Analysis,  Cr i ter ion-Ref erenced  Objective, 
and  Objectives  Hierarchy  Report  (#6) 

Goal  Analysis  Report  (#9) 

F-16  Pilot  and  Instructor  Pilot  Target  Population 
Study  (#13) 

F-16  Coursewares  and  Syllabi  Delivery  Schedule  (#24) 

F-16  Instructor/Course  Manager  Training  Requirements 
Report  (#25) 

F-16  Pilot  Media  Selection  (#26) 

Tasx  Listings  and  Criterion-Referenced  Objectives 
For  the  Instructor  Pilot  Training  Program  (#29) 

System  Operating  Manuals 

Data  Base  Update  Procedures  Report  (#10) 

Recoianenda ti ons  For  tne  F-16  Performance  Measurement 
System  (#14) 

F-16  Implaiaentation  and  Management  Plan  Report  (#18) 

(Quality  Control  Plans,  Procedures,  and  Rationale 
For  the  F-16  Pilot  Training  System  (#19) 

F-16  Instructional  System  Cost  Study  Report  (#21) 

F-16  Instructional  System  Basing  Concept  (#28) 

Summary  Reports 

A  User's  Guide  to  ttie  F-16  Training  Development 
Reports  (#34) 

F-16  Final  Report  (#35) 

y  • 

; 
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These  five  categories  of  documents  were  generated  in  a 
roughly  chronological  order.  First  a  series  of  instructional 
analysis  was  performed,  which  determined  the  content  and 
structure  of  the  F-16  Training  Program.  Second,  a  series  of 
methodology  reports  was  issued,  documenting  the  techniques 
utilized  in  the  analysis,  design,  and  development  of  the  program. 
Third,  the  data  bases  generated  during  analysis,  design  and 
development  were  recorded  for  future  use  in  maintaining  and 
modifying  the  training  system.  Fourth,  the  operating  manuals  or 
instructions  to  users  were  generated,  which  described  the  system 
and  its  operation.  Finally,  the  entire  history  and  output  of  the 
four  year  program  was  reviewed,  consolidated  and  recorded  in  the 
summary  reports.  A  more  detailed  description  of  the  major 
categories  follows: 


Problem  analysis,  review,  and  feasibility  studies:  These 

investigations  were  conducted  prior  to  the  development  of 
the  F-16  training  system  in  order  to  ensure  the  design  of  an 
effective  and  efficient  system.  These  reports  examine  the 
problems  to  be  solved,  the  resources  available  to  implement 
solutions,  and  the  primary  solution  alternatives.  Ttiey  go 
on  to  recommend  preferred  solutions,  such  as  heavy  reliance 
on  computer-assisted  and  computer  managed-instruction. 


Methodology  Reports:  The  second  group  of  reports  are  related  to 
ISD  procedures  such  as  analysis  of  tasks,  analysis  to  pro¬ 
duce  the  hierarchy  of  instructional  objectives,  selection  of 
instructional  media,  and  generation  of  syllabi.  In  conduct¬ 
ing  these  analytical  studies  the  project  staff  refined  or 
extended  some  existing  methods  and  procedures.  In  other 
cases  it  was  necessary  to  invent  new  procedures  appropriate 
to  specific  development  problems,  such  as  the  measurement  of 
incoming  student  abilities.  Separate  reports  have  been 
prepared  in  order  to  describe  the  methodological  innova¬ 
tions.  These  methodology  reports  also  describe  some  of  the 
problems  encountered  and  attempt  to  compare  and  contrast  the 
procedures  employed  with  existing  methodologies. 


Data  Base  Documents:  The  product  of  the  task  analysis  process 

was  an  inventory  of  F-16  pilot  tasks.  This  task  listing  was 
subsequently  used  to  generate  a  pool  of  criterion- 
referenced  objectives.  The  objectives  were  used  in  genera¬ 
ting  the  course  syllabus  and  also  the  media  selections.  Hie 
output  data  obtained  from  these  studies  constitute  the  data 
base  for  the  F-16  training  system.  The  contents  of  the  data 
base  are  stored  on  the  memory  disk  of  a  word  processor  where 
they  can  be  readily  updated  and  listed.  The  contents  are 
also  documented  in  the  data  base  reports. 
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Since  the  F-16  is  an  emerging  weapon  system,  it  is  necessary 
to  periodically  update  the  data  base  as  changes  in  the  air¬ 
craft  or  its  operational  procedures  are  initiated  by  the  Air 
Force.  Specific  procedures  for  keeping  the  data  base 
current  and  complete  are  described  in  project  report  no.  10, 
"Data  Base  Update  Procedures  Report”.  To  the  extent  that 
the  data  base  is  inadequate,  incomplete,  or  out  of  date  the 
training  system  will  be  deficient. 


System  Operating  Manuals;  The  F-16  training  system  is  a  complex 
assemblage  of  instructional  components  and  management  pro¬ 
cedures.  Some  of  the  management  procedures  are  actually 
subsystems  within  the  main  system.  The  "F-16  Implementa¬ 
tion  and  Management  Plan  Report”  presents  an  overview  of  the 
entire  system  and  prescribes  procedures  for  installing  the 
system.  The  other  system  operating  manuals  are  a  series  of 
handbooks  containing  and  utilizing  the  various  subsystems  in 
the  prescribed  manner.  Training  wing  commanders,  ISD  moni¬ 
tors,  OTD  team  members,  and  other  Air  Force  personnel 
responsible  for  administering  the  F-16  training  system 
should  become  very  familiar  with  these  documents  and  should 
refer  to  them  frequently. 

These  manuals  are  intended  to  be  working  documents.  They 
should  also  be  revised  as  changes  in  the  subsystems  are 
introduced  and  as  experience  with  the  prescribed  procedures 
accumulate . 


Summary  Reports;  These  two  documents  represent  after-the-fact 

description  of  the  processes  utilized  and  products  developed 
during  the  developnent  effort.  They  describe  the  state  of 
the  program  in  March  of  1981,  when  the  civilian  contractor 
completed  the  initial  development  work.  It  is  recommended 
that  the  OTD  team  issue  yearly  updates  to  the  Final  Report, 
documenting  the  changes  made  to  the  F-16  Training  System. 


2.0  EXECUTIVE  SUMMARIES 


This  section  contains  a  one  to  two  page  Executive  Summary  o 
each  report  in  the  P-16  Aircrew  Training  Development  series. 
These  summaries  are  included  in  the  front  matter  of  their 
respective  Reports  and  have  been  lifted  from  them  verbatim.  The 
size  and  scope  of  the  summary  does  not  reflect  the  magnitude  of 
its  corresponding  report,  which  may  contain  anywhere  from  10  to 
1,000  pages. 


2.1  Report  No.  1:  Program  Worlt  Plan 
EXECUTIVE  SUMMARY 


In  order  to  allow  precise  scheduling  and  monitoring  of  the 
F-16  Instructional  Systems  Development  effort,  a  detailed  work 
plan  for  the  project  is  outined.  This  work  plan  involves  nine 
phases,  each  of  which  is  divided  into  numerous  tasks.  Phase 
sequence  plans  show  the  tasks  to  be  performed  in  each  phase  of 
the  project  and  define  the  input/output  interdependencies  that 
exist  between  them. 

Phase  I  (1  task)  involves  a  review  of  the  contract  proposal 
and  a  briefing  to  USAF  and  contractor/subcontractor  personnel. 
Phase  II  (6  tasks)  involves  development  of  the  detailed  work  plan 
(this  document),  a  review  of  the  existing  F-16  task  analysis,  the 
design  of  data  collection  and  management  forms,  the  development 
of  a  task  and  goal  analysis  work  plan,  and  the  design  of  the 
format  for  criterion-referenced  objectives  (CROs)  and  tests  to  be 
used  in  the  training  program.  Phase  III  (13  tasks)  involves  the 
completion  of  a  task  analysis,  goal  analysis,  and  system  con¬ 
straints  analysis.  The  major  outcomes  of  these  analyses  are  the 
selection  of  tasks  for  training  and  the  development  of  CROs  and 
tests.  Phase  IV  (24  tasks)  represents  the  major  design  phase  of 
the  project  and  includes  course  and  syllabus  design,  media 
analysis  and  selection,  design  of  the  instructional  management 
and  quality  control  system,  and  definition  of  the  instructor/ 
course  manager  requirements.  Phase  V  (9  tasks)  involves  the 
development  and  production  of  the  courseware  (including  instruc¬ 
tor/course  manager  courses)  and  the  development  of  the  instruc¬ 
tional  management  system.  Phase  VI  (10  tasks)  involves  valida¬ 
tion  and  revision  of  the  training  system,  simulator  certifica¬ 
tion,  the  development  of  basing  concepts,  data  base  management, 
and  the  development  of  the  project  Final  Report. 

For  each  task  within  each  phase,  a  detailed  list  of  events 
is  provided  in  tabular  format  with  columns  for  personnel  and 
completion  dates.  This  allows  the  present  work  plan  to  be  used 
as  a  detailed  progress  monitoring  instrument  and  also  facilitates 
easy  revision  to  the  work  plan  during  the  course  of  the  project. 


2.2  Report  No.  2:  Previous  ISD  Program  Review 
EXECUTIVE  SUMMARY 


The  F-16  Instructional  Systems  Development  (ISD)  effort 
represents  the  most  comprehensive  application  of  ISD  principles 
and  procedures  for  pilot  training  yet  conducted  within  the  Air 
Force.  In  order  to  avoid  the  shortcomings  and  problems  of 
previous  Air  Force  applications  of  ISD,  a  careful  analysis  of 
past  Air  Force  ISD  programs  has  been  accomplished  to  determine: 
(1)  the  lessons  which  have  been  learned  that  relate  to  the  F-16 
efforts  and  (2)  those  problems  that  have  presented  the  most 
difficulty  in  the  application  of  ISD  technology  to  pilot 
training.  This  report  summarizes  the  results  of  that  analysis. 

Interviews  were  conducted  with  ISD  personnel  from  A-7,  A-10, 
F-15,  and  F-4  communities.  Conclusions  drawn  from  the  results  of 
the  interviews  are  as  follows: 

1.  Ensure  command  support  and  understanding  of  F-16  ISD 
programs  and  principles  is  maintained. 

2.  Man  ISD  teams  adequately  and  with  trained  ISD  specialists. 

3.  Document  all  F-16  efforts  including  rationales  for  all 
decisions  and  policies  for  use  in  future  programs. 

4.  Provide  sufficient  support  for  data  handling  and  access, 
i.e.,  a  dedicated  ISD  ADP  system. 

5.  Establish  a  joint  ISD/FLIT  team  to  coordinate  and  smooth  the 
way  from  Fighter  Lead-in  Training  to  Combat  Crew  Training 
(CCT)  squadrons. 

6.  Tactics  training  will  be  a  problem  area.  Experimentation 
and  use  of  other  fighter  aircraft  experience  will  be 
necessary. 

7.  Review  thoroughly  the  A-lO/F-15  ISD  experience  for  insight 
in  establishing  the  line  of  division  between  CCT  subjects 
and  continuation  training  subjects. 

8.  Special  attention  should  be  paid  to  alternate  training 
methods  to  substitute  for  the  unavailability  of  trainers  in 
the  initial  phases  of  the  F-16  training  project. 

9.  Objective  evaluation,  (CROs  and  CRTs) ,  must  be  developed  in 
close  cooperation  with  STAN/EVAL  personnel. 
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2.3  Report  No.  3:  Data  Collection  and  Management  Forms  Report 

EXECUTIVE  SUMMARY 

This  report  details  the  forms  developed  for  organizing  and 
storing  the  voluminous  ISO  data/information  generated  by  the  P-16 
training  project.  The  forms  are  designed  to  accommodate  continu¬ 
ally  changing  data  relative  to  hundreds  of  tasks,  instructional 
objectives,  and  production  elements.  Although  the  forms  are  the 
product  of  experience  in  manually  operated  systems,  they  are 
compatible  with  computer  based  data  systems.  The  formats  used 
have  been  changed  and  modified  as  necessary  as  P-16  ISD  pro¬ 
cedure.'  have  changed. 

This  report  presents  a  sample  and  description  of  the  forms 
used  in  the  analysis  and  design  phases  of  the  program,  that  is 
(1)  tasks  listing  and  objectives  hierarchy  analysis  (task  list¬ 
ing/objectives  hierarchy  worksheet  and  task  specification  work¬ 
sheet)  ,  (2)  media  selection  (hands-on  media  selection  forms, 
media  by  capability  matrices,  media  priority  by  segment  sheet, 
and  media  selection  tally  sneet) ,  (3)  personnel  management  (week¬ 
ly  time  summary  sheet) ,  (4)  production  management  (production 
management  forms,  weekly  progress  report,  and  production  tracking 
chart) ,  and  (5)  pre-implementation  evaluation  (student  attitude 
questionnaire)  . 

The  fact  tnat  the  forms  illustrated  and  described  were  de¬ 
signed  and  redesigned  during  the  P-16  project  is  not  of  concern, 
since  forms  are  management  tools  to  serve  the  thought  processes, 
and  not  vice  versa.  Tools  in  any  industry  are  designed/re¬ 
designed  to  meet  tne  cnanging  requirements  of  that  industry. 

Despite  their  susceptibility  to  change,  it  is  recommended 
that  great  care  be  taken  in  providing  the  data  called  for  by  the 
forms  because  they  are  the  product  of  experience  gained  from 
other  130  projects.  Also,  by  early  development  of  these  forms, 
data  collection  was  timely  and  systematic,  resulting  in  the 
savings  of  time  and  money. 

Data  bases  required  for  an  automated  ISO  management  system 
have  all  been  defined,  should  automation  be  implemented  after  the 
contractor  has  left  the  P-16  training  program. 
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2.4  Report  No.  4:  Review  of  Existing  F-16  Task  Analysis 

EXECUTIVE  SUMMARY 


A  task  analysis  is  a  very  critical  and  fundamental  component 
of  an  ISD  project  since  it  ultimately  determines  what  tasks  and 
contents  will  be  included  in  the  training  program.  The  existing 
task  analysis  for  the  F-16  pilot  compiled  by  General  Dynamics  was 
analyzed  in  terms  of  the  specific  requirements  of  the  F-16 
training  program.  Five  major  criteria  were  used: 

1.  Appropriate  level  of  detail. 

2.  Adequacy  of  coverage  of  job  tasks. 

3.  The  mission  orientation  (rather  than  equipment 
orientation)  of  the  analysis. 

4.  Appropriate  scope  of  behaviors. 

5.  Logical  consistency  of  task  breakdown. 

The  level  of  detail  and  scope  of  this  existing  task  analysis 
were  found  to  be  adquate.  The  coverage  of  job  tasks  was  found  to 
be  inadequate  and  in  need  of  revision.  The  mission  orientation 
of  the  existing  analysis  was  considered  to  be  weak  in  the  areas 
of  tactics,  premission  planning,  and  air-to-sur face  combat.  The 
logical  consistency  of  the  analysis  was  found  to  be  weak  and  in 
need  for  revision  in  the  areas  of  air  maneuvers  and  system  opera¬ 
tions. 

It  was  concluded  that  the  existing  F-16  task  analysis  would 
be  helpful  as  a  suggestive  tool,  but  additional  analysis  is 
required  to  provide  a  solid  foundation  for  the  F-16  ISD  effort. 
Areas  particularly  needing  attention  are  those  behaviors 
associated  with  cognitive  performance  (e.g.,  air-to-air  or  air- 
to-surface  combat,  mission  planning,  navigation,  etc.)  as  opposed 
to  equipment  operation. 
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2.5  Report  No,  5:  Deris/ation,  Formatting  &  Use  of  CROs  &  CRTs 

EXECUTIVE  SUMMARY 


An  integral  part  of  the  F-16  instructional  design  process 
involves  the  development  of  criterion-referenced  objectives 
(CROs)  and  criterion-referenced  tests  (CRTs) .  This  report 
defines  both  CRO  and  CRT  as  used  in  the  P-16  project.  It  also 
specifies  procedures  and  conventions  that  were  used  to  write 
them. 


There  are  several  benefits  associated  with  the  use  of  CROs 
and  CRTs.  By  following  the  procedures  described  in  this  report, 
a  team  of  minimally  trained  people  can  produce  a  clear  definition 
of  the  contents  of  a  training  program.  The  problems  of  not 
knowing  where  to  start,  guesswork,  and  confusion  are  eliminated 
when  this  systematic  approach  is  used. 

CROs  represent  specific  behavioral  statements  about  expected 
student  performance  after  the  completion  of  instruction.  The 
conditions  and  standards  of  acceptable  performance  are  also  part 
of  this  statement.  Specifying  instructional  outcomes  in  terms  of 
student  performance  has  a  number  of  advantages: 

1.  CROs  are  related  directly  to  actual  job  performance. 

2.  CROs  provide  a  focus  for  the  student  in  the  form  of  a 
statement  describing  what  he  should  be  able  to  do. 

3.  CROs  provide  a  source  of  feedback  to  the  student  by  offering 
him  a  chance  to  compare  his  performance  with  the  required 
performance,  which  in  turn  means  that  a  large  part  of  the 
instruction  becomes  self-guided. 

The  CRTs  are  a  logical  extension  of  the  CROs.  They  measure 
the  attainment  of  the  CROs.  Since  the  CROs  consist  of  actual  job 
performance  objectives,  the  CRTs  provide  the  instructor  and  the 
student  with  a  profile  of  the  student's  strengths  and  weaknesses 
on  job  performance.  CRTs  for  the  measurement  of  CROs  have  the 
following  minimal  characteristics. 

1.  A  description  of  the  environment  and  equipment  required  in 
the  test  setting. 

2.  A  description  of  the  problem  situation. 

3.  A  set  of  instructions  to  the  student  describing  the 
performance  expected. 
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4.  A  description  of  the  evaluator  of  the  behavior  to  be 
measured  or  noted. 

5.  A  set  of  evaluation  rules  for  rating  each  measurement  to 
determine  mastery. 

6.  A  method  or  form  for  the  evaluator  to  record  the  results  of 
the  measurements. 

7.  A  rule  for  combining  individual  measurements  in  a  task  or 
course  into  a  pass/fail  statement. 

In  summary,  CROs  and  CRTs  tell  all  personnel  involved  in 
pilot  training  just  exactly  what  should  be  taught,  what  should  be 
learned,  and  what  level  of  competence  is  expected. 
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2.6  Report  No.  6;  F-16  Task  Analysis,  CRO,  Objectives  Hierarchy 


EXECUTIVE  SUMMARY 

This  report  contains  the  P-16  pilot  training  task  listing, 
criterion-referenced  objectives  (CROs) ,  objectives  hierarchies, 
and  coursemap.  A  task  listing  is  the  logical  breakdown  of  a  task 
or  job  into  its  component  subtask.  For  instructional  purposes, 
each  of  these  subtasks  is  then  converted  into  a  CRO  complete  with 
conditions  and  standards  for  successful  performance.  The  in¬ 
terrelationship  of  the  CROs  is  identified  and  represented  in  a 
hierarchical  arrangement. 

For  example,  the  major  task  of  "performing  the  duties  of  an 
F-16  pilot"  was  divided  into  the  following  11  subtasks: 

1.  Premission  planning 

2.  Pretakeoff  procedures 

3.  Takeoff 

4.  Departure 

5.  Enroute  procedures 

6.  Air  refueling 

7.  Combat 

8.  Recovery 

9.  Landing 

10.  Post-flight  procedures 

11.  Mission  debriefing 

Each  of  these  subtasks  were  then  broken  down  into  smaller  per¬ 
formances.  For  instance,  under  premission  planning  such  tasks  as 
collect  weather  data,  collect  operations  data,  etc.  were 
identified.  These  performances  form  the  basis  of  the  CROs.  This 
reduction  in  task  complexity  provides  the  logical  rationale  for 
the  hierarchical  arrangement. 

All  tasks  relevant  to  the  F-16  training  program  are  listed 
in  this  report.  This  provides  the  foundation  for  all  subsequent 
instructional  design  and  development  activities. 


2.7  Report  No.  7;  Task  Analysis  Methodology  Report 

EXECUTIVE  SUMMARY 


A  well  done  task  analysis  reduces  waste  from  too  much 
instruction  while  at  the  same  time  ensures  important  skills  are 
taught.  Task  analysis  is  a  process  of  breaking  a  job  into  an 
inventory  of  component  tasks  called  a  task  listing.  The  task 
listing  must  provide  an  accurate,  comprehensive  list  of  all  tasks 
performed  on  a  job  in  order  for  a  training  program  to  include 
instruction  of  relevant  behaviors.  The  task  listing  forms  the 
foundation  from  which  every  major  activity  in  the  instructional 
development  process  follows.  Its  importance  is  therefore 
paramount . 

The  task  listing  procedure  used  in  the  F-16  project  involved 
the  following  steps: 

1.  Listing  of  all  responsibility  areas  for  a  given  job. 

2.  Listing  of  all  major  missions  within  each  responsibility 
area . 

3.  Dividing  missions  into  phases. 

4.  Listing  of  all  tasks  for  each  phase  by  means  of  a  mental 
"walk  through"  with  a  subject  matter  expert  (SME) . 

5.  Organizing  tasks  so  that  all  levels  of  tasks  are 
independent . 

6.  Identifying  all  unusual  conditions  in  tasks  which  may 
require  additional  training. 

7.  Examining  current  course  curricula,  technical  manuals, 
equipment,  etc.  for  possible  omissions. 

8.  Reviewing  task  listings  with  SME(s)  outside  of  the  analysis 
team. 

Since  turnover  in  personnel  involved  in  the  original  F-16 
task  analysis  can  be  expected,  the  methodology  used  for  the 
analysis  is  outlined  in  this  report.  The  ability  to  acquaint  new 
personnel  with  the  specific  steps  of  the  analysis  is  important 
for  several  reasons.  The  analysis  was  done  on  an  emerging 
weapons  system  and  changes  in  the  task  listing  can  be  expected. 
Personnel  must  know  how  to  incorporate  new  tasks  within  the 
existing  task  listing.  Also,  following  an  outlined  methodology 
decreases  the  chances  of  omissions  or  unnecessary  "reinventing 
the  wheel."  Informed  decisions  on  task  changes  and  listing  will 
lead  to  optimal  growth  of  the  F-16  training  program. 
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Because  the  task  listing  must  be  seen  as  an  evolving 
document,  the  secondary  purpose  of  this  report  is  to  suggest 
procedures  and  role  assignments  that  would  facilitate  the 
management  and  updating  of  the  task  listing.  To  ensure  proper 
maintenance  of  the  task  listing,  it  is  recommended  that 
organizational  roles  and  responsibilities  for  task  listing 
changes  be  clearly  defined  and  the  required  information  channels 
be  established  via  routine  procedures.  In  addition,  a  number  of 
resources  should  be  accessible,  such  as  word  processing, 
necessary  personnel,  and  reproduction  facilities. 
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2.8  Report  No.  8:  Objectives  Hierarchy  Analysis 
Methodology  Report 


EXECUTIVE  SUMMARY 


Objective  hierarchy  analysis  is  the  process  of  identifying 
the  specific  instructional  performances  students  must  achieve  in 
order  to  master  the  behaviors  indicated  in  a  task  listing.  The 
objectives  identified  in  the  hierarchy  analysis  should  include 
all  of  the  categories  of  behavior  required  to  prepare  students 
for  on-the-job  performance.  These  categories  include  verbal 
behaviors,  motor  behaviors,  and  intellectual  skills.  The 
objective  hierarchy  analysis  procedure  used  in  the  F-16  project 
included  the  following  steps; 

1.  A  task  is  selected. 

2.  Subtasks  are  listed. 

3.  Decision-making  behaviors  (classification  and  rule  using) 
are  identified. 

4.  Motor  control  behaviors  are  identified. 

5.  Verbal  support  (summarization)  requirements  are  identified. 

6.  Minor  tasks  are  grouped  for  instruction. 

7.  Dangerous  and  excessively  expensive  behaviors  are  scaled 
back. 

8.  Knowledge-base  requirements  are  determined. 

9.  Remaining  tasks  are  eliminated  (pruning). 

10.  A  new  task  is  selected  and  processing  begins  again. 

In  order  to  facilitate  the  updating  and  revision  of  the 
objectives  hierarchies,  the  maintenance  responsibilities  of 
various  ISD  personnel  and  the  related  support  requirements  are 
described . 


2.9  Report  No.  9:  Goal  Analysis 
EXECUTIVE  SUMMARY 


Goal  analysis  is  the  process  of  translating  instructional 
goals  into  observable  indicator  behaviors  which  can  be  taught  and 
measured.  In  order  to  develop  a  list  of  goals  for  the  F-16  pilot 
training  program,  a  literature  review  was  conducted  and  exper¬ 
ienced  F-4  pilots  were  interviewed  to  identify  the  attributes  and 
attitudes  characterizing  a  successful  combat  pilot.  From  these 
data,  a  list  of  eleven  major  goals  was  identified.  These  goals 
are ; 


1.  A  high  standard  of  preparation. 

2.  Excellent  physical  fitness. 

3.  Highly  sel  f- rel  ient . 

4.  Timely  and  decisive  in  actions. 

5.  Situational  awareness  during  flight. 

6.  Dete  iled  knowledge  of  enemy  tactics,  capabilities,  and 
weaki. esses . 

7.  Highly  flexible  and  resourceful. 

8.  Foresight. 

9.  Competitive  and  aggressive  in  combat. 

10.  Maintain  composure  in  combat. 

11.  Possess  a  high  degree  of  discipline. 

For  each  goal,  indicator  behaviors  and  character isitics  to 
be  incorporated  into  the  instructional  system  to  facilitate  these 
goals  were  identified.  This  information  will  be  important  in  the 
development  of  the  performance  measurement  system  as  well  as  the 
quality  control  of  the  F-16  training  project. 
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2.10  Report  No.  10:  Data  Base  Update  Procedures  Report 


EXECUTIVE  SUMMARY 


This  report  describes  procedures  for  updating  the  following 
five  major  ISD  data  bases: 

1.  Pilot  task,  analysis. 

2.  Goal  analysis. 

3.  Criterion-referenced  objectives  (CROs). 

4.  Objectives  hierarchies. 

5.  Program  reports. 

Tbese  data  bases  are  divided  into  two  types  which  require 
updating;  interdependent  data  bases  and  isolated  data  bases. 

The  interdependent  data  bases  include  the  task  listings, 
CROs,  and  objective  hierarchies.  Because  these  lists  are  highly 
interrelated  cnanges  in  one  listing  requires  cnanges  in  the  other 
two  lists.  As  an  example,  changes  in  the  heads-up  display  (HUD) 
of  tne  aircraft  might  require  that  all  HUD  related  tasks  are 
examined  for  updating.  Changes  in  the  task  listing  may  require 
changes  in  tne  objectives  hierarchy  and  the  CROs  involving  the 
HUD.  Thus,  changes  in  an  interdependent  data  base  require 
changes  in  related  data  bases. 

Isolated  data  bases  such  as  reports,  i.e.,  the  target 
population  study,  can  be  changed  and  updated  without  affecting 
other  data  bases.  Isolated  data  bases  are  independent  and 
changes  in  one  data  base  will  not  require  the  updating  for 
rewriting  of  otner  data  bases. 

Changes  to  these  data  bases  arise  from  two  sources;  external 
and  internal.  External  changes  derive  from  sources  outside  the 
ISD  process  such  as  equipment  modifications,  changes  in  tactics, 
or  different  student  entry  skills.  Internal  changes  are  those 
revisions  which  derive  from  the  ISD  process  itself  such  as  a  need 
for  changes  as  pointed  out  by  a  program  evaluation. 

The  process  of  changing  the  data  bases  starts  when  a  member 
of  the  contractor  staff  or  OTD  team  becomes  aware  of  the  need  for 
a  change  to  data  base  documents  or  reports.  A  hard  copy  of  the 
pertinent  data  is  obtained  by  that  person  and  proposed  changes 
are  inserted  into  these  printouts  and  the  appropriate  revision 
forms.  The  changes  are  reviewed  by  appropriate  personnel  and  if 
approved  all  areas  affected  by  the  change  are  noted  and  the 
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cnange  is  incorporated  into  the  appropriate  document/report  data 
bases.  The  procedure  described  is  a  general  overview  of  the  data 
update  procedures.  The  report  includes  specific  steps  for 
updating  both  interdependent  and  isolated  bases  as  well  as  the 
task  revision  and  report  revision  forms  to  be  used. 
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2.11  Report  No.  11;  Data  Automation  of  Task  and  Goal  Analysis 

EXECUTIVE  SUMMARY 


Development  report  no.  11  reviews  the  existing  data  automa¬ 
tion  system  for  task  and  goal  analysis  before  making  recommenda¬ 
tions  (1)  concerning  the  use/nonuse  of  automatic  data  processing 
(ADP)  in  support  of  task  and  goal  analysis  processing  and  (2) 
proposing  a  suitable  automated  system,  if  automation  is 
recommended : 

Two  possible  methods  of  utilizing  ADP  support  are  discussed: 
(1)  as  an  authoring  aid  and  (2)  to  assist  in  record  keeping, 
validation,  and  updating  of  task  and  goal  lists. 

Although  ADP  could  be  used  during  the  early  authoring  stage 
of  task  and  goal  analysis  by  providing  automatically  sequenced 
prompts  ("how  to  do  it"  displays),  it  is  not  recommended  for  two 
reasons . 

1.  The  interactive  nature  of  the  analysis  processes  (frequent 
reconsideration,  juggling,  and  rewriting  of  task/subtask 
hierarchies)  is  better  handled  on  a  large  working  surface 
where  alternatives  are  all  in  view  for  arrangement  and 
possible  modif iciation. 

2.  Subtle  judgements  are  included  in  analyses  which  are  best 
handled  by  experienced  training  analysis  specialists. 

ADP  is  well  suited  for  handling  the  accumulation,  storage, 
and  recall  of  the  hundreds  of  F-16  tasks  and  the  task  numbers, 
behaviors,  conditions,  standards,  and  CRO  data  for  each  of  the 
P-16  tasks.  Validation  reviews  and  updating  are  also  handled 
easily  by  ADP. 

The  existing  TAG  ADP  system  uses  the  Burroughs  550  computer 
system  and  is  described  in  TACM  50-300.  This  system  provides 
good  ADP  support  in  reducing  information  survey  time.  However, 
it  has  important  shortcomings  involving: 

1.  Card  input  time  and  storage:  All  program  input  is  recorded 
on  punched  data  cards.  This  is  a  cumbersome  system  because 
task  data  must  be  coded  and  formatted,  cards  must  then  be 
punched  by  a  qualified  keypunch  operator  and  hund  carried  to 
the  data  automation  unit.  Stored  cards  are  subject  to 
jumbling  and  require  a  substantial  staffing 

2.  Off-line  operation  turnaround  time;  Data  extraction  is  not 
rapid.  It  usually  requires  several  hours  because  the  user 
does  not  interface  directly  with  the  program.  The  user  must 
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submit  cards,  which  are  stacked/queued  to  await  processing 
and  the  final  printout. 

3.  Program  maintenance/user  sophistication:  The  present  system 
is  powerful,  but  effective  use  requires  a  project  staff 
member  with  a  good  working  knowledge  of  the  computer 
program. 

4.  The  lack  of  a  goal  analysis  program:  The  present  program 
analyzes  tasks  but  not  goals. 

Recommendation:  We  recommend  using  the  DEC  WPS-8  Word 
Processing  System.  This  system  is  now  used  at  the  F-16  project 
office.  The  system  is  very  cost  effective  and  offers  immediate 
support  for  task  and  goal  analysis.  Although  it  lacks  some  of 
the  formatting  abililities  of  the  present  system,  it  has  a 
powerful  sorting  ability  and  also  other  major  advantages  i.e., 

(1)  incorporates  micro-electronics  for  si ze- reduction ,  (2)  is  a 
self-contained  microprocessing  system  with  a  printer,  therefore, 
eliminating  cumbersome  punch  cards,  (3)  interface  is  direct 
through  a  keyboard  and  screen,  (4)  work  is  performed  on-line  so 
turnaround  time  is  immediate  (5)  storage  is  on  convenient 
flexible  diskettes,  each  of  which  holds  over  3,000  punchcards  (6) 
operation  is  simple,  and  doesn't  require  a  program  language  code. 
Secretaries  usually  develop  the  basic  skills  in  one  week  of 
concentrated  practice/training. 

We  feel  the  benefits  of  the  DEC-WPS-8,  many  of  which  are 
readily  available,  will  provide  the  USAF  F-16  program  with  a 
modern,  responsive,  and  flexiblle  ADP  system.  Not  only  is  the 
system  inexpensive  but  it  should  result  in  extensive  savings  of 
time  and  money. 


2.12 


Report  No  12:  Training  Management  System 


Needs  Analysis 
EXECUTIVE  SUMMARY 


Within  TAG  the  complexity  of  the  weapons  systems  is  increas¬ 
ingly  rapid.  Thera  is  also  a  corresponding  increase  in  the 
training  requirements  for  these  systems.  In  order  to  develop  and 
maintain  an  efficient  and  effective  training  program  within  real¬ 
istic  constraints  of  manpower  and  training  resources,  the  manage¬ 
ment  system  should  be  responsive  to  the  following  needs: 

1.  Lifetime  maintenance  of  the  training  program. 

2.  Standardization  of  the  approach  to  modifying  the  training 
system. 

3.  A  method  of  exploring  varying  program  alternatives. 

4.  Quick  response. 

5.  Early  detection  of  problems. 

6.  Flexibility  to  meet  changing  needs. 

7.  Self-monitoring  capabilities. 

8.  Capability  to  deal  with  large  amounts  of  data. 

9.  Establ isnment  of  a  historical  data  base. 

10.  Redundant  management  of  information. 

11.  Operation  of  the  system  at  the  user  level  of  training 
managers,  instructors,  and  students. 

Existing  training  management  systems  were  reviewed  and  were 
found  to  be  deficient  in  terms  of  their  capabilities  of  meeting 
tne  above  needs.  For  example,  none  of  the  systems  have  provided 
adequate  support  for  the  initial  ISD  effort,  nor  were  they  found 
to  be  adequate  for  long  term  support  of  an  ISD  project. 

Based  on  the  identified  needs  and  the  information  gained  by 
examining  previous  and  existing  training  management  systems,  a 
system  is  described  which  has  the  potential  capability  of  meeting 
the  needs  of  the  F-16  training  project.  This  system  is  a  mix  of 
procedures,  personnel,  support  capabilities,  and  organizational 
structures  tnat  will  facilitate  the  planning,  management,  and 
implementation  of  the  F-16  program  over  the  life  cycle  of  the 
aircraft . 
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2.13  Report  No  13:  F-16  Pilot  and  Instructor  Pilot 

Target  Population  Study 


EXECUTIVE  SUMMARY 

A  target  population  study  provides  a  detailed  description  of 
incoming  student  characteristics  and  focuses  on  prior  training 
and  entry-level  skills  relevant  to  the  training  program  being 
designed.  The  F-16  target  population  study  involved  the 
following  major  activities. 

1.  A  review  of  existing  student  population  studies. 

2.  Interviews  With  training  personnel  and  graduates. 

3.  An  incoming  student  inventory  using  the  F-16  pilot  task 
1 isting. 

The  data  obtained  from  F-4  and  F-15  profiles  indicates  that 
the  initial  F-16  IPs  will  be  considerably  more  experienced  than 
IPs  later  in  the  program.  This  suggests  that  the  initial  IP 
syllabus  will  need  to  be  changed  to  reflect  this  characteristic. 
Data  obtained  from  the  F-4  and  F-15  conversion  and  course  popula¬ 
tion  suggests  a  similar  difference  between  initial  and  subsequent 
students,  particularly  in  the  area  of  tactical  knowledge. 

Student  pilots  for  the  B  (basic)  F-16  course  will  normally 
have  graduated  from  the  USAP  Undergraduates  Pilot  Training  (UPT) 
program  and  the  Fighter  Lead-in  Training  (FLIT)  program.  The 
selection  of  students;  their  educational  background,  physical 
characteristics,  psychological  background,  and  flight  experience; 
and  the  training  syllabi  of  UPT/FLIT  are  examined.  This  data  was 
used  to  develop  a  skill  profile  of  the  incoming  B  students  to  be 
used  in  other  F-16  instructional  design/development  activities. 
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2.14  Report  No  14:  Recommendations  for  the  F-16  Performance 

Measurement  System 


EXECUTIVE  SUMMARY 


The  major  purpose  of  this  report  is  to  present  "state  of  the 
art"  recommendations  for  developing  the  P-16  PMS.  Before  the 
recommendations  are  made,  theoretical  and  practical  concerns  that 
a  PMS  must  address  are  presented.  Theoretical  issues  such  as 
reliability,  validity,  and  the  rule  linking  measurement  to  grades 
are  seen  as  fundamental  to  the  measurement  process.  But  these 
concepts  must  be  implemented  within  real  world  constraints. 

Since  regulations,  for  example  TACR  50-31,  determine  the 
structure  and  content  of  performance  measurement  practice,  these 
documents  were  reviewed  and  their  guidelines  evaluated  in  terms 
of  existing  systems  within  the  Air  Force.  The  A-10,  F-15,  and 
F-4  systems  were  reviewed.  The  direction  provided  by  TACR  50-31 
ranges  from  precise  prescriptions  to  very  broad  guidelines.  Tbis 
has  both  good  and  bad  points.  On  the  one  hand,  the  broadness  of 
the  guidelines  allows  flexibility  for  individual  training  systems 
to  adapt  to  local  needs.  However,  the  lack  of  specifics  on 
critical  matters  like  grade  interpretation,  the  remediation 
process,  and  the  function  of  gradeslips  are  seen  as  ambiguous 
areas  that  might  lead  to  confusion. 

The  final  section  of  the  report  presents  a  proposal  for  the 
F-16  PMS.  Although  specific  proposals  are  made  on  the  tools  to 
be  used,  personnel  involved,  and  record  keeping  incorporated  in 
the  PMS,  this  summary  will  present  only  the  highlights  of  the 
proposed  system.  The  major  innovations  are  as  follows: 

1.  Use  of  automated  academic  tests  and  quizzes. 

2.  Concern  with  higher  level  evaluation  rather  than  rote 
memorization  (where  appropriate). 

3.  Use  of  a  comprehensive  student  progress  report. 

4.  Improved  gradeslips  that  will  identify  student  strengths  and 
weaknesses . 

5.  Procedures  for  proficiency  advancement. 

6.  Measurement  of  both  part  and  whole  tasks. 
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7.  IP  instruction  in  performance  measurement. 

8.  Automated  record  keeping. 

At  this  time,  the  final  decisions  on  what  the  PMS  will  look 
like  have  not  been  made.  The  PMS  that  will  be  used  for  the  F-16 
training  program  will  be  described  in  report  number  18,  F-16 
Implementation  and  Management  Plan  Report. 


2.15  Report  No.  15:  Prog ram/System  Constraints  Analysis  Report 

EXECUTIVE  SUMMARY 


The  development  and  implementation  of  a  training  system  is 
carried  out  in  the  presence  of  certain  constraints  which  impact 
the  effectiveness  or  efficiency  of  the  training.  These  con¬ 
straints  include  operating  policies  and  regulations,  existing 
facilities  and  equipment,  characteristics  of  the  student  popula¬ 
tion,  and  personnel  availability.  The  task  of  the  contractor 
team  was  to  investigate  and  identify  constraints  likely  to  affect 
the  F-16  instructional  system  design  and  describe  system  design 
restrictions  imposed  by  them.  Such  information  will  be  used  by 
the  contractor  team  in  designing  the  P-16  Instructional  system  to 
accomodate  those  constraints  identified,  and  by  USAP/TAC  agen¬ 
cies  in  an  effort  to  mitigate  the  effeccs  of  these  constraints 
whenever/wherever  possible.  Such  efforts  should  result  in 
increased  communication  between  the  contractor  team  and  USAP/TAC 
agencies  in  coordinating  possible  changes. 

The  constraints  operating  in  the  P-16  environment  were 
categorized  as  due  to: 

1.  Limited  data. 

2.  Existing  regulations. 

3.  Resource  limitations. 

4.  Existing  programs. 

5.  Given  inputs. 

6.  Environmental  factors. 

7.  Expected  system  changes. 

Por  each  constraint,  specifics  have  been  identified  and  the 
data  source,  probable  Impact  on  system,  possible  actions,  and 
implications  are  discussed. 

Some  constraints  identified  such  as  weather,  range  availa¬ 
bility,  and  air  space  are  "hard"  and  must  be  accomodated  by  the 
training  system.  Other  constraints  such  as  USAP/TAC  regulations 
and  policies,  trainer  delivery  schedules,  or  student  flow  could 
be  changed  to  be  more  compatible  with  the  training  program. 
Finally,  some  constraints  such  as  facilities  and  media  available 
are  within  the  scope  of  the  training  system  and  can  be  modified. 
As  a  result  of  this  study  analysis,  the  F-16  ISO  effort  has  been 
able  to  better  plan  the  training  program  within  existing  system 
constraints . 
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2.16  Report  No.  16:  A  Stud/  of  Media  Production  and 
Reproduction  Options  for  the  F-16  Project 


EXECUTIVE  SUMMARY 

The  on-time  production  of  instructional  materials  is  a 
critical  factor  in  the  successful  implementation  of  the  F-16 
training  system.  It  is  estimated  that  the  total  production 
requirements  for  the  B,  IP,  and  Continuation  Courses  will  be  405 
workbooks,  405  slidetape  sets,  and  90  videotapes.  In  addition, 
another  1,012  pages  of  printing  will  be  required  for  handbooks, 
tests,  and  checklists.  These  estimates  were  based  on  the  number 
of  segments  suggested  by  the  task  listing  document  and  certain 
assumptions  regarding  the  nature  of  the  workbooks  and  tapes 
required . 

The  production  and  reproduction  capabilities  located  at  Hill 
AFB,  Ogden,  Utah  were  examined  to  determine  if  the  production  and 
reproduction  needs  of  the  F-16  project  could  be  met  there.  Pro¬ 
duction  refers  to  the  developnent  of  original  instructional 
materials  (both  prototype  and  final)  and  the  preparation  of 
reproducible  masters.  Reproduction  is  the  duplication  of 
camera-ready  masters,  slides,  audio  and  videotape  masters  for 
distribution  to  users. 

Tne  Hill  AFB  printing,  pnoto  lab,  and  television  center 
capabilities  are  seen  as  adequate  to  support  the  reproduction 
requirements  of  tne  F-16  project  as  presently  estimated. 

In  the  area  of  production ,  it  appears  that  the  printing  shop 
does  not  nave  tne  capacity  to  handle  the  volume  of  F-16  require¬ 
ments  for  pint  layout  and  design.  However,  printing  can  provide 
eril ar jement/reduction  capability  for  tne  proportion  of  photo¬ 
static  copies  to  be  used  in  workbook  paste-up  in  addition  to  a 
moderate  amount  of  printing  samples  and  proof  copies.  A  woricing 
reldtionsnip  witn  Detachment  8,  which  operates  the  Ogden  Air 
Logistics  Center  Television  Center,  is  recommended,  provided  that 
some  adjustment  of  priorities  is  made  and  if  adequate  provision 
can  be  made  for  technical  and  instructional  direction  by  tne 
contractor.  The  graphics  unit,  as  presently  staffed,  could 
support  tne  F-16  project  in  tne  production  of  graphics.  Certain 
constraints  and  considerations  in  the  production  of  F-16  photo- 
grapny  mane  it  imperative  tnat  pnotographers  be  contractor 
supplied  although  it  would  be  advantageous  to  establish  a 
worxing  agreement  with  the  Hill  AFB  photo  lab  for  additional 
support  in  surge/crisis  situations. 

Major  concerns  in  the  establishment  of  this  interface 
between  government-supplied  and  contractor-supplied  agencies  and 
personnel  are  accountability,  communication,  and  control.  The 


contractor  is  accountable  for  instructional  and  technical  quality 
and  the  cost  and  timeliness  of  the  product.  Problems  can  result 
in  the  communication  of  necessary  directions  and  specifications 
between  producing  agencies.  The  concern  over  control  deals  with 
the  loss  of  control  on  the  part  of  the  contractor  over  those  who 
are  producing  a  contract  deliverable  for  which  the  contractor  is 
responsible 


2.17  Report  No  17:  Computer  Managed  Instruction  for  tne  F-16 

Training  Program 


EXECUTIVE  SUMMARY 


The  computer  managed  instructional  (CMI)  system  required  for 
the  F-16  training  program  must  include  components  for  automatic 
testing,  record  keeping,  student  scheduling,  inventory  tracking, 
and  instructional  prescriptions.  The  testing  components  must 
provide  facilities  for  test  grading,  diagnostic  testing,  and  test 
security,  as  well  as  adequate  student  feedback  capability.  It 
must  also  permit  the  evaluation  of  instructional  effectiveness 
and  facilitate  course  revision.  The  student  records  component 
must  encompass  bibliographic  information,  training  performance 
data,  report  generation  of  records,  and  multiple  sorting  capabil¬ 
ities.  The  scheduling  component  must  provide  the  capabilities 
for  tne  optimized  scneduling  of  student  learning  activities  and 
training  resources.  This  scheduling  capability  must  exist  at  the 
levels  of  syllabus  and  weekly  and  daily  timetables.  Tne  inven¬ 
tory  component  must  provide  short-  and  long-term  information  on 
all  resource  utilization  and  availability,  including  personnel, 
equipment,  facilities,  and  supplies.  The  prescriptive  component 
must  provide  tne  capability  to  generate  individualized  student 
learning  schedules  based  upon  student  performance  data,  as  well 
as  training  resource  availability  and  utilization. 

Five  existing  instructional  systems  which  could  serve  as  the 
basis  for  the  F-i6  CMI  system  are  surveyed.  For  each  of  the 
systems  a  functional  description  and  examples  of  input/output  are 
piovided.  Tne  five  systems  studied  were  AFORMS,  TICCIT,  VTS, 

Navy  CMI,  and  AIS.  None  of  the  five  systems  surveyed  could 
satisfy  ail  of  the  capabilities  required  in  the  functional  speci¬ 
fications  for  the  F-16  training  environment,  and  the  limitations 
of  each  are  mentioned. 

A  cost  benefit  analysis  considered  the  general  cost  benefits 
applicable  to  the  F-16  environment.  These  include  savings  in  the 
amount  of  training  time  required,  more  effective  utilization  of 
training  resources,  improved  quality  control,  increased  standard¬ 
ization  and  flexibility  in  the  training  system,  and  greater  field 
deployability  and  expandibility.  Specific  cost  considerations 
for  the  F-16  program  were  made  by  extrapolating  from  data  on  the 
F-4/F-15  training  programs.  The  specific  cost  savings  projected 
on  tnis  basis  were  a  time  reduction  of  two  weeks  in  course  dura¬ 
tions,  or  25  student  salaries  for  the  training  period;  10  man 
years  of  instructor  time  per  course;  and  17  full-time 
administrative  support  personnel. 
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The  conclusion  of  the  study  considered  the  five  systems 
surveyed  in  the  report  in  terms  of  functional  capabilities, 
operational  status,  remote  capability,  training  status,  and  cost 
factors.  Although  any  of  the  five  system  could  b  used  as  a  basis 
for  the  F-16  CMI  system,  it  was  recommended  that  VTS  is  presently 
the  most  suitable  system,  because  it  a)  meets  nearly  all  of  the 
functional  requirements,  b)  is  relatively  inexpensive,  c)  is 
basically  a  "stand-alone”  system,  d)  is  currently  in  use  for 
aircrew  training  and  is  an  approved  military  training  device,  and 
e)  has  sufficient  scope  to  support  a  full-scale  training  system. 


2.18  Report  No  18:  F-16  Implementation  and  Management 

Plan  Re po r  t 


EXECUTIVE  SUMMARY 


This  report  presents  plans  and  procedures  for  implementing, 
using,  and  maintaining  the  F-16  pilot  training  system.  The  goals 
of  the  report  are  to:  (1)  describe  the  F-16  pilot  training  system 
and  the  administrative  context  in  which  it  is  designed  to 
function,  (2)  recommend  specific  policies  and  procedures  for  im¬ 
plementing  and  operating  the  F-16  training  system,  and  (3)  to 
coordinate  and  tie  together  all  of  the  plans  and  documents  pro¬ 
duced  during  the  development  of  the  training  system. 

In  addition  to  specifying  and  describing  the  roles  of  per¬ 
sonnel  required  to  operate  the  F-16  training  system,  the  forms 
needed  to  store  and  record  data  generated  by  this  system  are  also 
presented.  Specific  explanations  about  who  fills  out  the  form, 
how  it  is  filled  out,  and  its  disposition  after  completion  is 
g iven. 

The  report  concludes  by  listing  and  describing  the  major 
criteria  that  can  be  used  to  evaluate  the  adequacy  of  any  in¬ 
structional  system  and  how  the  system  described  in  this  docu¬ 
ment  measures  up  to  each  criterion.  This  proposed  management 
system  represents  a  step  in  the  right  direction,  but  it  does  not 
significantly  impact  costs  as  it  is  believed  a  computer  managed 
system  would. 


2.19  Report  No  19:  Quality  Control  Plans,  Procedures, 
and  Rationale  for  the  P-16  Pilot  Training  System 


EXECUTIVE  SUMMARY 

Quality  control  is  a  fundamental  component  of  the  ISO 
process.  In  order  to  develop  and  maintain  an  efficient  and 
effective  training  system,  it  is  essential  to  monitor  the  system 
and  make  improvements  on  the  basis  of  feedback  data  obtained  from 
evaluation.  Quality  control  procedures  are  applicable  at  three 
main  stages  in  the  development  and  implementation  of  a  training 
system : 

1.  The  developmental  evaluation  state  (preimplementation). 

2.  'fte  operational  monitoring  state  (implementation). 

3.  Graduate  evaluation  (follow-up). 

Common  shortcomings  of  previous  evaluation  efforts  in 
military  ISD  are: 

1.  Too  narrow  in  scope. 

2.  Inappropriate  data  collected. 

3.  Incorrect  analysis  or  interpretation  of  data. 

4.  Results  are  not  utilized  to  rectify  weaknesses  in  system. 

The  quality  control  plan  for  the  F-16  project  attempts  to 
avoid  these  shortcomings.  The  developmental  procedure  will 
involve  small-scale  tryouts  for  the  individual  instructional  seg¬ 
ments  followed  by  larger  group  tryouts  with  mul ti- segment 
materials.  The  operational  monitoring  stage  will  involve  evalua¬ 
tion  of  data  produced  by  the  performance  measurement  system  (PMS) 
which  includes  routine  and  systematic  data  collection  procedures. 
The  graduate  evaluation  stage  will  involve  three  types  of  data  to 
be  collected  for  each  graduating  class: 

1.  Task  specific  proficiency  ratings. 

2.  Results  of  STAN/EVAL  check  ride,  simulator  check,  and  final 
examination . 

3.  Graduate  evaluation  questionnaires  and  structured  interviews 
with  graduates. 

To  assist  in  these  three  stages  of  quality  control,  evaluat¬ 
ion  specification  checks  are  provided  at  six  major  checkpoints  in 
the  training  cycle. 
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2.20  Report  No  20:  F-16  Instructional  System 

Cost  Study  Report 


EXECUTIVE  SUMMARY 


Media  selection  is  the  process  of  minimizing  the  costs  due 
to  media  acquisition,  use,  or  maintenance  while  maximizing  the 
likelihood  of  consistent  instructional  effectiveness.  Media 
selection  for  aircrew  training  involves  two  major  categories  of 
media : 

1.  Media  intended  for  use  in  learning  centers,  ranging 
from  workbooks  to  computer-assisted  instruction. 

2.  Simulator  or  training  devices  which  are  intended  to 
provide  "hands-on”  practice. 

Factors  which  must  be  considered  in  the  media  selection 
process  include  cost,  ability  of  a  device  to  implement  specific 
instructional  strategies,  motivating  capabilities,  and  the  pop¬ 
ularity  of  the  media  device  with  the  student  population.  Some  of 
the  constraints  to  be  considered  in  the  F-16  environment  include 
the  necessity  of  premature  media  decisions,  the  late  arrival  of 
training  devices,  the  varying  availability  of  media  cross  train¬ 
ing  sites,  the  exisiting  media  production  facilities,  the  tenta¬ 
tive  nature  of  the  instructional  content,  the  required  mixture  of 
type  "A"  and  "B"  aircraft,  and  the  varying  availability  of  flight 
time  across  training  sites. 

The  media  selection  process  begins  with  a  complete  list  of 
the  criterion-referenced  objectives  (CROs)  from  the  objectives 
hierarchy  analysis.  These  objectives  are  divided  into  academic 
(i.e.,  learning  center  learning  activities)  and  training  device 
categories,  and  then  classified  in  terms  of  taxonomic  levels. 
Instructional  strategies  appropriate  for  each  level  of  learning 
required  are  then  specified. 
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For  those  objectives  identified  as  requiring  training 
devices,  additional  media  selection  steps  are  required.  It  is 
necessary  to  identify  behavior  elements  contributing  to  task 
difficulty.  These  elements  are  mapped  into  corresponding  levels 
of  training  devices  ranging  from  low  fidelity,  non- interact ive 
devices  (e.g.,  cardboard  mock-ups)  to  high  fidelity,  highly  in¬ 
teractive  devices  (e.g.,  operational  flight  trainers).  The  F-16 
aircraft  is  assigned  only  when  complete  fidelity  is  instruction- 
ally  necessary.  The  possible  configurations  for  each  training 
device  are  specified,  along  with  the  time  requirements  associated 
with  each  configuration.  The  cost  of  each  configuration  is  then 
estimated  and  the  best  configuration  selected. 

The  media  selection  process  is  a  dynamic  one  and  the  pro¬ 
cedures  described  extend  over  the  lifetime  of  the  F-16  training 
system.  To  aid  in  the  standardization  and  documentation  of  the 
media  selection  process,  a  set  of  forms  are  provided  which  assist 
in  the  data  collection  and  decision-making  steps. 
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2.  21 


Report  No.  21:  F-16  Instructional  System  Cost 

Study  Report 


EXECUTIVE  SUMMARY 


All  training  program  designs  must  eventually  be  trans¬ 
lated  into  a  set  of  resources  required  to  support  training 
activities  and  the  costs  associated  with  these  resources.  Re¬ 
source  requirements  vary  depending  upon  the  type,  amount,  and 
complexity  of  the  training  to  be  conducted  and  generally  cover  a 
very  broad  spectrum  of  support  needs.  The  detailed  assessment  of 
training  requirements  is  the  result  of  a  training  support  re¬ 
quirements  analysis  (TSRA)  which  can  then  be  used  to  generate 
specific  cost  figures. 

The  F-16  instructional  system  represents  a  complex  system  in 
which  the  training  resources  required  range  from  equipment  such 
as  aircraft,  simulators,  and  trainers  to  personnel  such  as  flight 
instructors,  academic  instructors,  course  managers,  and  adminis¬ 
trative  support  personnel.  The  spectrum  of  resources  also 
includes  media  devices,  training  materials,  training  facilities, 
and  subtraining  systems  for  the  preparation  of  system  personnel. 
The  purpose  of  this  report  is  to  provide  a  tool  for  calculating 
resource  support  requirements  and  associated  costs  of  the  F-16 
pilot  and  instructor  pilot  instructional  system  based  upon  the 
data  available  at  any  point  in  system  operation. 

Recomputation  should  be  carried  out  using  this  cost  study  at 
any  time  the  effects  of  a  real  or  contemplated  change  to  the 
input  conditions  are  of  interest  to  system  managers.  Such 
changes  include  system  design  changes  (e.g.,  heavier  use  of 
self-study,  elimination  of  personnel  through  job  consolidation, 
or  changes  in  management  procedures),  syllabus  requirements 
changes  (e.g.,  through  addition  or  subtraction  of  aircraft 
sorties,  through  the  addition  of  new  training  devices,  or  through 
the  addition  of  new  academic  content),  and  student  load  changes. 

It  is  not  the  purpose  of  this  report  to  provide  a  model  for 
operational  or  logistic  planning.  Because  the  model  does  not 
include  a  time  component,  it  is  not  useful  for  resource  sched¬ 
uling  or  resource  allocation  purposes.  Furthermore,  the  model 
assumes  a  steady  state  training  system  with  smooth  instructor  and 
student  flow.  Almost  all  input  variables  are  averages.  This 
means  that  the  model  does  not  compute  resources  needed  for  peak 
loads  or  "surges"  on  the  training  system  or  the  effects  of  class 
build-ups,  student  attribution  or  resource  unavailability.  For 
these  types  of  operational  factors,  a  training  system  management 
model  is  needed. 
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2.22  Report  No  22:  Recommendations  for  F-16  Operational 
Flight  Trainer  (OFT)  Design  Improvements 


EXECUTIVE  SUMMARY 


Because  the  training  development  contractor  has  been 
involved  early  in  the  life  cycle  of  the  aircraft,  the  F-16  pro¬ 
ject  provides  an  opportunity  for  instructional  developers  to 
provide  input  to  the  design  of  training  devices  to  be  developed 
for  the  F-IS  training  system.  Some  of  the  capabilities  already 
incorporated  in  the  design  of  the  F-16  Operational  Flight  Trainer 
(OFT)  such  as  freeze,  performance  replays,  and  automated  measure¬ 
ment  are  highly  desirable  instructional  features.  In  addition, 
it  is  recommended  that  capabilities  be  provided  by  grade  sheet 
production,  simulator  setup  and  off-line  debriefings,  mission 
status  and  look-ahead,  a  self-instructional  capability,  and  a 
"help"  capability  for  instructors.  Two  problems  with  the  physi¬ 
cal  arrangement  of  the  OFT  for  training  demonstrations  were  iden¬ 
tified.  A  series  of  principles  were  specified  for  the  formatting 
of  instructional  displays.  A  number  of  recommendations  were  made 
regarding  the  design  of  the  OFT  for  simulating  aircraft  malfunc¬ 
tions.  Finally,  recommendations  were  made  in  terms  of  designing 
the  OFT  to  accomodate  growth  and  changes  over  the  F-16  training 
system  lifespan. 

While  the  involvement  of  the  ISD  team  in  the  simulator 
design  process  has  been  helpful  in  identifying  design  deficien¬ 
cies,  participation  prior  to  simulator  procurement  in  future 
projects  would  result  in  training  devices  which  are  more  closely 
aligned  to  the  requirements  of  the  training  system. 
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2.23  Report  No  23:  F-16  Instructional  Sequencing 

Plan  Report 


EXECUTIVE  SUMMARY 


Ihis  report  details  the  rationale  and  procedure  for 
sequencing  (syllabus  building)  the  pilot  and  instructor  pilot 
(IP)  syllabi  of  the  F-16.  Principles  of  instructional  sequenc¬ 
ing  are  presented  and  descriptions  given  of  the  F-16  sequencing 
process  and  syllabus  revision  procedures. 

Development  of  a  syllabus  involves  evaluation  of  (1)  student 
entry  level  skills,  (2)  training  and  instructional  goals/objec¬ 
tives,  (3)  availability  of  training  resources,  and  (4)  the  behav¬ 
ior  expected  of  graduates.  "nie  development  of  a  syllabus  is 
based  upon  a  set  of  learning  principles.  Three  examples  are: 

1.  Speed  of  forgetting  (forgetting  begins  almost  immediately 
unless  there  is  prompt  testing  or  rapid  use  of  new 
information)  . 

2.  Interference  (learning  of  some  tasks  interferes  with  the 
learning  of  other  tasks  when  they  possess  similar  but 
confusing  differences.  Remembering  that  final  approach 
speeds  of  155,  140,  and  150  kts  are  characteristic  of  the 
F-4,  F-16,  and  A-7  respectively  is  more  difficult  than 
remembering  final  approach  speeds  of  155  and  60  for  the  F-4 
and  piper  cub  respectively) . 

3.  Transfer  (prior  learning  of  some  skills  enhances  the 
learning  of  another  skill.  Skill  at  proper  scanning  of 
instruments  for  IFR  flight  gained  in  one  aircraft  will 
facilita^'e  the  same  task  in  another  aircraft). 

The  syllabus  structure  must  be  linked  to  the  criterion- 
referenced  objectives  as  well  as  the  career  performance  goals  of 
the  students.  To  generate  a  syllabus  a  model  must  exist  which 
describes  the  necessary  steps  to  acquire  a  skill.  For  the  F-16 
this  involves: 

1.  Academic  training;  Base  preparation  for  task  execution 
(workbooks,  tape/slides,  videotapes,  lectures,  discussion 
groups,  and  tutorials). 

2.  Practice  in  limited  simulated  performance  environment; 
Simulators,  mockups,  CPT,  dead  aircraft. 

3.  Practice  in  real  world  performance  environment  or  near  real 
world  environment.  OFT,  WST,  actual  aircraft. 
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Specific  procedures  for  generating  the  F-16  syllabus  involve: 


1.  Definition  of  the  mastery  models  and  terminal  performance 
tests . 

2.  Determination  of  the  desired  levels  of  performance  for  each 
task  (given  by  the  CRTs). 

3.  Determination  of  the  syllabus  structure  based  upon  aircraft 
as  the  only  training  device. 

4.  Determination  of  non-aircraft  training  devices  to  preserve 
aircraft  utilization. 
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2.24  Report  No  24:  F-16  Coursewares  and  Syllabi 

Delivery  Schedule 


EXECUTIVE  SUMMARY 

Ibis  document  performs  the  two-fold  purpose  of  (1)  describ¬ 
ing  the  schedule  whereby  all  course  materials  will  be  delivered 
to  the  Air  Force  and  (2)  presenting  the  B  course  syllabus  that 
will  be  utilized  to  schedule  the  presentation  of  those  course 
materials  to  the  RTU  students.  It  represents  a  historical  docu¬ 
ment  only,  reflecting  a  delivery  schedule  proposed  in  1979  and 
the  latest  version  of  the  syllabus  available  in  early  1981.  It 
was  significant  as  a  deliverable  item  in  1979,  as  it  documented 
for  the  Air  Force  precisely  what  it  was  getting  and  when  it  would 
be  arriving. 

This  document  describes  the  development  of  four  separate 
courses  with  unique  syllabi:  the  B/C  course,  TX  course,  IP 
course,  and  continuation  training  course.  This  original  con¬ 
tractor  requirement  was  subsequently  reduced  to  include  a  B 
course  only,  so  the  IP  and  TX  syllabi  included  represent  primar¬ 
ily  the  efforts  of  the  OTDT,  while  the  continuation  training 
requirement  was  deleted  altogether. 


2.25  Report  No  25:  F-16  Instructor/Course  Manager 

Training  Requirements  Report 


EXECUTIVE  SUMMARY 


A  task  listing  indicates  what  instructional  activities  will 
be  performed  by  F-16  academic  instructors  (or  course  managers) 
and  flight  instructors  (both  simulator  and  aircraft).  The 
academic  instructor  tasks  include  authoring  activities  as  well  as 
the  implementation  and  administration  of  instructional  activities 
in  the  learning  center.  Flight  instructor  tasks  include 
conducting  and  supervising  training  sessions  for  all  interactive 
and  non- interactive  training  devices  and  actual  training  sorties 
in  the  F-16  "A"  or  "B"  aircraft.  These  later  tasks  involve 
demonstration  and  evaluation  of  basic  aircraft  operation, 
emergency  procedures,  and  tactical  maneuvers. 
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2.26 


Report  No  26:  F-16  Pilot  Media  Selection 


EXECUTIVE  SUMMARY 


This  report  contains  three  types  of  information.  The  F-16 
task  listing  represents  a  detailed  and  organized  collection  of 
all  the  major  tasks  which  an  F-16  fighter  pilot  must  perform. 

Ttie  criterion-referenced  objectives  represent  the  testing 
situations  that  correspond  to  each  major  task  and  ensure  that 
students  are  accountable  for  these  tasks.  The  objectives 
hierarchies  break  down  each  task  in  the  task  list  into  a  se¬ 
quence  of  academic  prerequisite  objectives  that  are  designed  to 
lead  to  mastery  of  the  task  itself. 

The  document  is  divided  into  11  sections,  corresponding  to 
11  mission  subtasks,  from  "1.1  Perform  Premission  Planning"  to 
"1.11  Perform  Mission  Debriefing".  Each  of  these  sections  is 
further  subdivided  into  tasks,  criterion- ref erenced  objectives, 
and  objectives  hierarchies.  Thus,  the  major  sections  are  se¬ 
quenced  according  to  the  major  phases  of  flight,  while  the 
subsections  present  an  orderly  elaboration  of  tasks  into 
criterion- ref erenced  objectives.  Ths  report,  like  other  data 
base  documents,  is  an  evolving  record  of  decisions  which  must  be 
periodically  updated  to  remain  useful. 


2.  27  Kapo  r  L  No  27  : 


F-IS  Instructional  S/sta^n  Desijn  Concept 
EXECUTIVE  SUMMAHY 


f'le  purpose  of  tnis  report  is  to  present  trie  alter nati>/a 
desi'jns  for  the  F-16  Repiaceinent  Training  Unit  (RTJ)  constraints  and 
costs  iiivolv/ed  in  their  use.  Thoujh  the  systein  is  capable  of 
administering  several  courses  of  instruction  at  once,  the  costs 
reported  are  tnose  for  trainin^j  B/C  course  students  only. 

While  the  primary  purpose  of  a  training  system  is  to  deliver 
instruction,  there  are  additional  functions  k^nich  must  be  per¬ 
formed  in  order  to  ensure  that  the  instructional  process  remains 
maximally  effective.  Many  of  these  auxilliary  functions  do  not 
need  to  be  performed  when  only  one  set  of  students  is  to  be 
trained;  however,  long  term  effectiveness  is  severly  degraded  if 
they  are  ignored  wnen  designing  a  system  that  will  continue  for  a 
number  of  classes.  The  complete  set  of  functions  recommended  for 
inclusion  in  the  F-16  training  system  is  specified  in  Attach¬ 
ment  I,  and  is  organised  in  terms  of  subsystems. 

Five  basic  options  are  described  in  terms  of  their  assets, 
limitations,  and  costs.  These  options  are  as  follows: 

Option  I  (Manual  System)  A  systematically  designed  and  up¬ 
dated  system  where  tne  academic  portion  of  tne  course  is 
taught  largely  through  existing  instructional  media  (print¬ 
ed  material,  audio  tapes  and  slides,  supplemented  occasion¬ 
ally  by  videotape  presentations).  Mediated  itistruction  is 
self-paced  to  allow  review  of  material  or  restudy  of  por¬ 
tions  of  lessons  any  time  the  student  wishes.  Use  of 
academic  instructor  time  to  present  information  is  mini¬ 
mized  so  that  instructors  can  maximize  the  time  spent  deal¬ 
ing  with  individual  problems  and  guestions.  Use  of  training 
devices  and  aircraft  is  carefully  integrated  with  self-paced 
and  classroom  instruction  so  that  the  student  can  derive 
maximum  benefit  from  equipment  use  through  complete  and 
thorough  preparation.  Centralized  record  Keeping  and  ad¬ 
ministrative  functions,  as  well  as  distributed  scheduling 
functions,  are  carried  out  using  form  driven  procedures  sucn 
that  lesson  updates,  data  base  changes,  etc.,  are  handled  as 
efficiently  as  is  possible  without  automated  data  processing 
support . 

Option  II  (Basic  CM I-augmen ted  System)  Inis  option  adds 
automated  data  processing  support  for  centralized  adminis¬ 
trative  and  distributed  scheduling  functions.  By  keeping 
track  of  individual  student  progress  as  well  as  student 
response  to  individual  lessons  on  line,  individualized 
syllabus  options  are  more  readily  generated  and  specific 
instructional  deficiencies  are  more  readily  identified  and 
corrected.  In  this  way,  the  mediated,  self-paced  instruc- 
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tional  plan  detailed  in  Option  I  is  more  fully  utilized 
while  support  personnel  spend  less  time  performing  clerical 
duties  and  have  more  time  to  plan  for  systematic  improve¬ 
ments  . 

Option  III  (Basic  CMI,  CAI,  and  part-task  Training  System). 
Tliis  option  adds  the  capability  of  computer-assisted  in¬ 
struction  (CAI)  and  computer  controlled  part-task  hands-on 
training  to  the  computer  management  package  whose  impact  is 
described  in  Option  II.  These  new  capabilities  allow 
learner  control  at  the  learning  strategy  level  so  that  in 
addition  to  individual  syllabus  prescription,  students  also 
benefit  from  the  ability  to  select  the  method  of  studying 
each  lesson  that  best  suits  their  individual  cognitive 
style.  Since  CAI-mediated  lessons  allow  the  unique  self¬ 
tailoring  in  instructional  presentation  by  many  students 
simultaneously,  individual  differences  in  learning  style  are 
better  accommodated.  In  addition  to  a  greater  degree  of 
individualization  in  academic  instruction,  the  addition  of 
computer  controlled  part-task  hands-on  training  allows  the 
system  to  provide  more  and  better  training  in  basic  psycho¬ 
motor  tasks,  providing  greater  amounts  of  monitored  hands- 
on  instruction  while  increasing  instructor  availability  for 
supervising  higher  order  hands-on  training  in  more  complex 
training  devices. 

Option  IV  (Full  CMI,  CAI,  and  Integrated  part-task  Train¬ 
ing  System  with  Automated  Performance  Resource  Capability). 
Under  this  option,  the  capabilities  described  in  Option  III 
are  expanded  to  increase  the  amount  of  instruction  that  can 
be  placed  under  learner  control  with  the  addition  of  an 
automated  performance  measurement  system.  Such  a  system 
allows  the  measurement  and  recording  of  student  performance 
at  any  point  in  his  indiviudal  program  of  instruction.  This 
in  turn  allows  more  accurate  diagnosis  of  individual  problem 
areas  and  instructional  deficiencies.  Such  a  performance 
measurement  system  need  not  be  restricted  to  academic  in¬ 
struction  only,  but  can  also  encompass  the  measurement  and 
recording  of  student  performance  on  computer  controlled 
hands-on  training. 

Option  V  (Addition  of  Automated  Performance  Measurement) . 
This  option  expands  the  performance  measurement  system  to 
incorporate  automated  performance  measurement  of  in-flight 
tasks  through  the  use  of  Air  Combat  Maneuvering  Instru¬ 
mentation.  This  allows  precise,  objective  measurement  of 
in-flight  performance,  as  well  as  student  self-evaluation  by 
means  of  the  recording  and  playback  features  of  the  system. 
By  adding  this  capability,  the  precision  of  diagnosis  of 
individual  problems  in  actual  job  performance  is  greatly 
enhanced,  allowing  for  maximally  useful  remediation. 
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2.28  Report  No  28:  F-16  Instructional  System  Basing  Concept 

EXECUTIVE  SUMMARY 


The  purpose  of  this  report  is  to  present  a  concept  for  the 
basing  of  F-16  Replacement  Training  Unit  (RTU)  training  sites  and 
a  mechanism  for  determining  the  configuration  and  training 
resource  requirements  at  each  site.  The  basing  concept  is 
presented  first,  defined  by  the  location  at  which  instructional 
system  functions  are  carried  out,  either  in  centralized  or 
distributed  form.  A  resource  calculation  tool,  called  the 
Training  Support  Requirements  Analysis  (TSRA)  ,  is  given  in  the 
form  of  a  step-by-step  job  aid  usable  by  all  levels  of  TAC 
training  management. 

The  scope  of  this  report  covers  the  basing  concept  from  the 
standpoint  of  instructional  system  function  and  efficiency  of 
system  operation.  Factors  related  to  operational  squadron  and 
wing  economics  and  concepts  of  weapon  employment  are  not  dealt 
with  here,  since  those  considerations  are  beyond  the  scope  of 
training  analysis.  In  making  basing  decisions  it  is  assumed  that 
TAC  will  consult  this  and  other  data  and  that  all  factors  will  be 
weighed  together  in  due  proportion. 

The  general  basing  concept  for  the  F-16  training  system  is 
defined  in  terms  of  system  functions.  Attachment  I  contains  a 
list  of  the  functions  which  the  F-16  training  system  is  designed 
to  execute.  Some  functions  are  directly  involved  in  the  instruc¬ 
tion  of  students  and  must  be  carried  out  in  a  distributed  fash¬ 
ion,  with  those  functions  being  executed  at  each  training  site. 
Other  functions  not  directly  related  to  instruction  of  students 
but  nonetheless  critical  may  be  carried  out  in  a  centralized 
fashion  at  one  location  within  the  system  (presumably  by  an 
Operations  Training  Development  (OTD)  team) . 

The  TSRA  worksheets  included  in  Section  3.0  allow  Air  Force 
personnel  to  determine  the  exact  requirements  for  Instructor 
personnel  (flight,  simulator  and  training  device,  classroom  and 
learning  center),  support  personnel  (instructor  and  student 
support,  as  well  as  learning  center  operators) ,  office  space  and 
furnishings,  learning  center  media  equipment,  training  device 
equipment,  learning  center  facilities  (study  area,  checkout  area, 
learning  center  operator's  office,  storage  area,  class¬ 
rooms,  training  device  space,  and  student  lounge)  and  training 
device  personnel  and  facilities. 
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2.29  Report  No.  29:  Task  Listings  and  Criterion-Referenced 
Objectives  for  the  Instructor  Pilot  F-16  Training  program 

EXECUTIVE  SUMMARY 


This  report  contains  two  lists:  The  IP  course  task  list  and 
the  IP  course  CROs .  The  task  list  represents  a  comprehensive 
inventory  of  the  major  tasks  an  F-16  pilot  should  be  able  to 
perform,  while  the  CROs  represent  the  major  testable  subtasks,  or 
performance  objectives  he  must  master  to  demonstrate  proficiency 
in  the  major  tasks.  The  CROs  are  derived  from  the  task  list 
through  hierarchical  analysis,  and  are  used  to  generate  the 
cr i terion- ref erenced  tests  (CRTs)  that  control  student  progress 
through  the  F-16  training  system. 
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2.30  Report  No  30:  F-16  Training  System  Media  Report 


EXECUTIVE  SUMMARY 


This  document  lists  the  32  academic  media  types  and  20 
simulation  media  types  selected  for  possible  inclusion  in  the 
F-16  training  program,  and  provides  a  detailed  description  of 
each  in  terms  of  its  instructional  features,  training  capability, 
life  cycle  costs,  hardware  and  software  characteristics,  man¬ 
power  requirements  and  facilities  requirements.  This  infor¬ 
mation  is  presented  in  the  form  of  matrix,  tables,  which  list 
media  characteristics  and  indicate  the  extent  to  which  each 
medium  possesses  those  characteristics. 

This  historical  document  was  produced  not  only  to  document 
the  F-16  media  pool,  but  as  a  preliminary  input  to  the  instruc¬ 
tional  media  selection  process.  The  52  media  could  not  be 
selected  and  assigned  in  either  the  cost  benefit  analysis  or 
media  analysis  without  first  characterizing  them  in  terms  of  the 
variables  listed  earlier.  Once  this  process  was  complete,  the 
media  were  utilized  in  the  instructional  media  selection  process 
(see  Development  Report  No.  20,  F-16  Media  Selection  and  Utiliza¬ 
tion  Plan  Report,  and  Development  Report  No.  31,  F-16  Training 
Media  Mix)  and  would  have  been  utilized  in  a  detailed  cost 
benefit  analysis,  had  that  requirement  not  been  cancelled  by  the 
contracting  officer. 


2.31  Report  No  31:  P-16  Training  Media  Mix 


EXECUTIVE  SUMMARY 


The  media  selection  process  utilized  in  this  program  is 
documented  in  three  separate  reports.  Report  No.  20,  "F-16  Media 
Selection  and  Utilization  Plan  Report",  describes  the  initial  31 
step  procedure  designed  to  select  instructional  media  for  various 
categories  of  objectives.  Report  No.  31,  "F-16  Training  Media 
Mix",  describes  the  15  step  procedure  actually  employed  on  the 
project.  Report  No.  26,  "F-16  Pilot  Media  Selection",  provides 
the  raw  data  used  in  the  selection  process.  Taken  together,  they 
provide  a  detailed  description  of  the  rationales  employed,  pro¬ 
cedures  followed,  data  generated,  recommendations  made,  and  media 
mix  utilized. 

Media  selection  consists  of  assigning  instructional  media  to 
groups  of  objectives  in  such  a  way  as  to  maximize  training  effec¬ 
tiveness  while  minimizing  training  costs.  All  objectives  are 
sorted  into  one  of  two  major  categories:  academic  objectives  and 
hands-on  objectives.  Both  sets  are  then  processed  along  parallel 
channels,  whereby  the  requirements  of  each  objective  are  computer 
matched  to  the  capabilties  of  each  medium.  This  produces  a 
prioritized  list  of  media  to  match  each  objective.  These  objec¬ 
tives  are  then  grouped  into  lessons,  trainer  sessions,  or  sorties 
according  to  commonalities  of  both  content  and  medium.  Later, 
during  the  syllabus  building  process,  these  lessons,  sessions, 
and  sorties  are  sequenced  into  a  functional  curriculum. 
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2.32  Report  No  32:  F-16  Training  Media  Support  Requirements 


EXECUTIVE  SUMMARY 


Training  media  support  requirements  are  those  personnel, 
facilities,  equipment,  and  materials/supplies  that  must  be 
purchased,  installed,  and  maintained  in  order  to  operate  any 
given  instructional  medium.  These  requirements  may  be  viewed 
costs  to  be  considered  before  deciding  to  procure  a  given  med 
or  media  mix,  or  resources  that  must  be  secured  and  in  place 
before  an  instructional  system  can  open  its  doors  to  students 
In  either  case,  they  represent  "hidden  costs"  that  have  not 
always  been  adequately  considered  before  establishing  an 
instructional  system. 

Section  1.0  describes  the  purpose  of  the  document  as  well  as 
the  general  format  and  approach.  Section  2.0  provides  a  compre¬ 
hensive  list  of  training  media  support  requirements,  including  a 
description  of  each  resource.  Section  3.0  lists  all  of  the 
potential  training  media,  both  academic  and  performance,  and 
provides  tables  which  specify  the  support  requirements  of  each. 
Section  4.0  lists  general  considerations  to  be  weighed  when 
evaluating  the  support  requirements  of  various  media. 


as 

ium 
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2.33  Report  No  33:  F-16  Training  Media  Constraints 

and  Limitations 


EXECUTIVE  SUMMARY 

The  purpose  of  this  report  is  to  list  and  discuss  the  con¬ 
straints  and  limits  of  the  various  instructional  media  employed 
in  P-16  training,  with  the  purpose  of  targeting  these  limita¬ 
tions  for  future  upgrading.  The  five  major  limitations  include: 
(1)  the  limited  number  of  training  devices  available,  (2)  the 
unavailability  of  certain  types  of  training  devices,  (3)  the  in¬ 
ability  to  impact  the  design  of  training  devices  because  of  the 
exceptionally  long  procurement  cycle,  (4)  the  use  of  less  than 
optimal  devices,  and  (5)  the  limited  range  of  media  available  for 
implementation  in  the  F-16  training  system. 

The  report  identifies  four  conditions  that  acted  to  pre¬ 
dispose  the  media  selection  process  to  the  constraints  identified 
above.  First,  the  F-16  syllabus  had  to  be  constructed  to  meet 
preexisting  aircraft  use  level  expectations  instead  of  allow¬ 
ing  instructional  considerations  to  drive  the  use  level.  Second, 
major  training  devices  were  designed  and  procured  before  specific 
training  uses  for  those  devices  were  generated.  Third,  no  in¬ 
structional  design  expertise  was  employed  during  the  initial 
stages  of  training  device  design  and  procurement.  Finally, 
adequate  funds  and  command  attention  were  not  directed  to  the 
procurement  of  major  F-16  training  devices. 

Based  upon  these  constraints  and  limitations,  the  report 
makes  six  specific  recommendations:  (1)  establish  and  execute  a 
plan  for  system  growth  that  will  eventually  minimize  these  con¬ 
straints,  (2)  design  simulated  practice  environments  an  scenarios 
that  take  fullest  advantage  of  the  capabilities  of  existing 
training  devices,  (3)  maintain  and  update  the  data-based  syllabus 
created  during  the  development  phase  and  utilize  feedback  from 
evaluations  to  adjust  the  syllabus,  (4)  examine  the  training 
functions  allocated  to  each  training  device  to  consider  possible 
reassignment  to  other  media,  (5)  employ  state-of-the-art  training 
techniques  to  increase  the  efficiency  of  training  device  utiliza¬ 
tion,  and  (6)  carefully  monitor  training  device  use  and  availa¬ 
bility  to  ensure  the  fullest  use  of  training  device  time. 
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2.34  Report  No  34:  A  User's  Guide  to  the  F-16 
Training  Development  Reports 


EXECUTIVE  SUMMARY 


Hie  F-16  Aircrew  Training  Development  Project  is  thoroughly 
documented  by  a  series  of  35  individual  development  reports. 

This  report  describes  the  organization  of  the  report  series  and 
provides  a  one  page  executive  summary  of  each  report.  While  the 
final  report.  No.  35,  is  designed  to  summarize  the  four  year 
development  effort  in  a  single  document,  report  No.  34  is  de¬ 
signed  to  index  the  report  series  for  those  readers  who  desire 
additional  information  concerning  a  particular  event  or  phase 
within  the  program. 
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2.35  REPORT  NO  35:  FINAL  REPORT 


EXECUTIVE  SUMMARY 


This  report  outlines  the  major  activities  of  the  F-16  Aircrew 
Training  development  Project/  a  four-year,  two  and  one  half 
million  dollar  Instructional  Systems  Development  effort  sponsored 
by  the  Tactical  Air  Command  to  design  a  comprehensive  training 
program  consist  of  a  5  1/2  month  Basic  Course,  a  2  1/2  month 
Transition  Qjurse,  and  a  35  volume  report  series  which  describes 
the  methodology  used  to  develop  those  courses. 

The  program  consisted  of  six  phases,  which  provide  the 
overall  organization  for  the  report.  The  project  is  reviewed  on 
a  phase  by  phase  basis,  the  lessons  learned  from  each  phase  are 
reviewed,  the  lessons  learned  from  the  overall  project  are 
reviewed,  and  conclusions  and  recommendations  are  provided. 

Phase  I  lasted  only  two  weeks  and  required  a  single 
deliverable  item,  the  Phase  I  Review.  During  this  period  the 
contractor  and  government  representatives  held  general  discus¬ 
sions  and  reviewed  in  detail  the  70  CDRL  Item  Descriptions.  This 
Phase  was  completed  a.®  planned. 

Phase  II  was  to  last  four  weeks,  allowing  the  development  of 
a  detailed  work  plan  for  the  entire  program.  The  contractor  also 
reviewed  previous  ISO  efforts  (e.g.,  A-7,  A-lO,  F-4  and  F-15)  , 
design  the  data  collection  and  management  forms  to  be  used  during 
the  program,  review  the  General  Dynamics  task  listing  for  applic¬ 
ability  to  the  training  program,  and  develop  formats  for  docu¬ 
menting  bothe  criterion-ref erenced  objectives  and  criterion- 
referenced  tests.  This  proved  to  be  more  difficult  an  assignment 
than  originally  predicted,  and  Phase  II  ended  with  only  5  of  the 
9  assigned  deliverables  complete. 

Phase  III  was  the  first  really  substantial  phase,  spanning  3 
1/2  months  and  requiring  delivery  on  13  contract  items.  Task 
analysis,  goal  analysis  and  objectives  hierarchy  analysis  were  to 
be  conducted  and  periodically  updated.  Target  population  studies 
for  the  various  F-16  courses  were  to  be  conducted,  as  were  the 
development  of  the  initial  criterion-referenced  objectives  and 
criterion-referenced  tests.  The  performance  measurement  system 
plan  would  be  devised,  the  program/ system  constraints  analyzed, 
and  the  printing  and  reproduction  facilities  identified  and 
emplo  yed . 


Phase  IV  was  to  be  the  first  of  4  year  long  phases,  requiring 
29  CDRL  items,  twice  as  many  as  any  other  phase.  It  represented 
the  completion  of  the  analysis  phase  of  the  program,  and  included 
sensitivity  analysis,  cost  benefit  analysis,  media  selection, 
recommendations  for  a  data  automation  system,  the  development  of 
a  performance  measurement  system,  and  the  development  of  planning 
docunents  for  course  design  and  course  management.  During  this 
phase  the  project  moved  form  Luke  AFB,  Arizona,  to  Hill  AFB, 

Utah . 

Phase  V  was  to  be  12  months  long,  requiring  the  delivery  of 
only  9  CDRL  items.  The  analysis  and  design.  Phases  would  now  be 
wares,  syllabi,  management  system,  program,  and  the  continuation 
training  plan. 

Phase  VI  was  to  be  the  course  implementation  phase,  again 
lasting  12  months.  The  Conversion  Course  and  Instructor  Pilot 
Course  would  begin,  and  those  course  materials  would  be  revised 
based  on  student  performance.  The  continuation  program  would  be 
developed,  to  be  implemented  along  with  the  Basic  Course  during 
the  next  phase.  Unfortunately,  the  class  date  for  the  first 
Basic  Course  was  March  1980  and  lesson  production  for  the  Basic 
Course  had  not  begun  until  August  of  1979.  But  the  Air  Force 
modified  the  contract  at  the  end  of  Riase  V,  moving  the 
Conversion  Course  and  Instructor  Pilot  Course  into  Phase  VII. 

Phase  VII  was  too  be  the  second  implementation  phase,  when 
both  the  Basic  Course  and  Continuation  Training  program  would 
begin.  The  revision  and  validation  of  all  courses  would  continue 
throughout  the  phase.  But  again,  the  contract  was  by  now  quite 
out  of  sync  with  the  original  phasing  plan.  The  Basic  Course  had 
already  been  implemented  in  I^tase  VI,  a  full  10  months  before  it 
was  originally  planned  for.  Continuation  training  was  still  a 
controversial  subject  at  TAC  headquarters,  and  the  OTD  teeun  was 
unable  to  provide  the  contractor  with  clear  guidance  as  to  how  to 
proceed.  Consequently,  the  Air  Force  decided  to  extend  Phase  VI 
an  additional  six  months  to  complete  all  work  it  actually  wanted 
to  complete,  and  simply  eliminate  Phases  VII  and  VIII.  Phase 
VIII  was  originally  to  last  only  3  months  anyway,  consisting  of 
update  and  revision  to  all  courses  and  the  drafting  of  the  Final 
Report.  It  was  reasoned  that  since  the  Basic  Course  had  begun  10 
months  early,  it  would  have  its  year  long  revision  cycle  by  March 
1981.  So  Phase  VI  was  expanded  to  accomodate  all  the  work 
programmed  for  Phases  VI,  VII  and  VIII  that  the  Air  Force  was 
still  interested  in.  This  involved  the  revision  of  the  B  Course, 
the  final  update  of  all  35  Project  Reports,  and  the  authoring  of 
the  Fi  nal  Repo  rt . 

After  discussing  each  phase  in  depth,  the  report  concludes 
with  a  section  on  lessons  learned  and  recommendations  to  future 
developers  of  weapons  systems  that  may  help  in  their  planning 
efforts.  Also,  the  report  contains  three  appendices.  The 
Statement  of  Work,  Evaluation  Data,  and  sample  course  materials 
are  included. 
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